Snapshots in Hadoop Distributed File System

The ability to take snapshots is an essential functionality of any file system, as snapshots enable system administrators to perform data backup and recovery in case of failure. We present a low-overhead snapshot solution for HDFS, a popular distributed file system for large clusters of commodity servers. Our solution obviates the need for complex distributed snapshot algorithms, by taking advantage of the centralized architecture of the HDFS control plane which stores all file metadata on a single node, and alleviates the need for expensive copy-on write operations by taking advantage of the HDFS limited interface that restricts the write operations to append and truncate only. Furthermore, our solution employs new snapshot data structures to address the inherent challenges related to data replication and distribution in HDFS. In this paper, we have designed, implemented and evaluated a fast and efficient snapshot solution based on selective-copyon-appends that is specifically suited for HDFS like distributed file systems.